Remote rendering of three-dimensional images using virtual machines

ABSTRACT

Remote rendering of three-dimensional images using virtual machines includes using a hypervisor executing on a physical computer to allocate exclusive and direct access to a graphics processing unit in the physical computer, to a first virtual machine. An agent executing on a second virtual machine intercepts three-dimensional draw commands generated by a three-dimensional application and forwards the intercepted draw commands to a rendering agent executing on the first virtual machine. The rendering agent then transmits the intercepted draw commands to the graphics processing unit for rendering upon which the graphics processing unit renders a three-dimensional image from the draw commands. The rendering agent obtains the rendered image from the graphics processing unit and forwards the image to the second virtual machine. Upon receiving the rendered image, the second virtual machine transmits the rendered image to another remote, physical computer where the rendered image is displayed to a user.

RELATED APPLICATIONS

This U.S. patent application claims priority to U.S. Provisional PatentApplication Ser. No. 61/241,420, filed on Sep. 11, 2009, the disclosureof which is considered part of the disclosure of this application and isherein incorporated by reference in its entirety.

FIELD OF THE DISCLOSURE

This application generally relates to rendering graphics using agraphics processing unit. In particular, this application relates toremote rendering of three-dimensional images in a virtualizationenvironment.

BACKGROUND OF THE DISCLOSURE

In many instances, hypervisors allow multiple operating systems executedby virtual machines to run simultaneously on the same physical hardware.These hypervisors can execute on top of an operating system, or canexecute directly on the physical hardware of a computer (e.g. bare-metalhypervisors.) In some instances a virtual machine can execute aparavirtualized operating system or a non-paravirtualized operatingsystem. Paravirtualized operating systems are typically operatingsystems that have been modified to execute within a virtualizedenvironment, e.g. the LINUX operating system, and can access hardwarecomponents directly. Non-Paravirtualized operating systems are typicallyoperating systems that have not been modified to execute within avirtualized environment, e.g. the WINDOWS operating system, and cannotaccess hardware components directly.

Rendering three-dimensional graphics typically requires a great deal ofresources and in many embodiments requires directly accessing a graphicsprocessing unit (GPU). In virtualized environments where multiplevirtual machines may require access to the graphics processing unit atany one time, the GPU is virtualized by the hypervisor and access to thevirtualized GPU may be managed by a control program. Three-dimensionalapplications that require rendering three-dimensional graphics,typically underperform in such virtualized environments because thethree-dimensional applications are not provided with direct andexclusive access to the GPU. Non-paravirtualized operating systemstypically require applications generating three-dimensional drawingcommands to access a GPU through a set of interfaces and drivers. Thus,users of the three-dimensional application may experience significantdelay in the rendering of three-dimensional images because thethree-dimensional application does not have direct access to the GPU.

Virtualizing the GPU can be one solution for overcoming the problemsassociated with executing three-dimensional applications within thecontext of a non-paravirtualized operating system. While the virtualizedGPU allows virtual machines to share access to a physical GPU, thevirtual GPU does little to ensure that three-dimensional applicationshave exclusive access to the physical GPU or to remedy the direct accessproblems posed by the non-paravirtualized operating systems. Further,implementing a virtual GPU often requires the installation and executionof a virtual GPU designed for the provided physical GPU.

There exist technologies that remedy the problems associated withrendering three-dimensional graphics in a virtualized environment. Thesetechnologies, however, often function for only a select fewthree-dimensional graphics libraries, e.g. OpenGL. For otherthree-dimensional graphics libraries, e.g. Direct3D, no such technologyexists. Furthermore, many of the technologies are only compatible withparavirtualized operating systems and not with non-paravirtualizedoperating systems.

There exist other technologies that remedy the problems associated withrendering three-dimensional graphics in a virtualized environment bytransmitting three-dimensional graphics commands to a remote computerable to render the graphics. This remote computer can in some instancesprovided the direct GPU access required to quickly and efficientlyrender the three-dimensional images. Upon rendering the graphics, theremote computer can transmit the rendered graphics back to the virtualmachine executing the three-dimensional application. Still othertechnologies provide a software graphics driver that provides access toa subset of the functionality provided by an available graphicsprocessing units. Each of these solutions fails to leverage thefunctionality of the physical GPU and may introduce latency into thegraphics processing process.

SUMMARY OF THE DISCLOSURE

In one aspect, described herein are methods and systems for remotelyrendering three-dimensional drawing commands generated bythree-dimensional applications executed by virtual machines executingnon-paravirtualized operating systems. A hypervisor executing on a firstphysical computer executes a first virtual machine and a second virtualmachine. The hypervisor allocates direct access to a graphics processingunit of the first physical computer to a non-paravirtualized operatingsystem executing on the first virtual machine. An agent executing on thesecond virtual machine intercepts three-dimensional draw commandsgenerated by a three-dimensional application that executes on the secondvirtual machine. The intercepted three-dimensional draw commands areredirected to a rendering agent that executes on the first virtualmachine. A graphics processing unit then renders an image from thethree-dimensional draw commands in response to receiving thethree-dimensional draw commands from the rendering agent. In response toreceiving the rendered image from the graphics processing unit, therendering agent forwards the rendered image to the agent executing onthe second virtual machine. In response to receiving the rendered image,the second virtual machine transmits the rendered image to a secondphysical computer over a communication channel established between thefirst physical computer and the second physical computer.

In some embodiments, the second virtual machine executes anon-paravirtualized operating system. In other embodiments, a remotingagent executing on the second virtual machine transmits the renderedimage to the second physical computer.

In other embodiments the rendering agent transmits the three-dimensionaldraw commands to the graphics processing unit of the first physicalcomputer in response to receiving the three-dimensional draw commandsfrom the second virtual machine. In still other embodiments, therendering agent transmit an instruction to render the three-dimensionaldraw commands along with the three-dimensional draw commands.

The second virtual machine, in some embodiments, receives a request froman application executing on the second physical computer for applicationoutput generated by the three-dimensional application executing on thesecond virtual machine. In some embodiments, the rendered image istransmitted to the second physical computer in response to receiving theapplication output request from the second physical computer.

The hypervisor, in some embodiments, allocates direct access to thegraphics processing unit to the first virtual machine by preventing thesecond virtual machine from accessing the graphics processing unit.

The second virtual machine, in some embodiments, requests access to thegraphics processing unit upon intercepting the three-dimensional drawcommands. The agent executing on the second virtual machine thenredirects the intercepted three-dimensional draw commands to the firstvirtual machine in response to receiving a notification from thehypervisor denying the second virtual machine access to the graphicsprocessing unit.

BRIEF DESCRIPTION OF THE DRAWINGS

The following figures depict certain illustrative embodiments of themethods and systems described herein, in which like reference numeralsrefer to like elements. These depicted embodiments are to be understoodas illustrative of the disclosed methods and systems and not as limitingin any way.

FIG. 1A depicts embodiments of network environments that provide remoteaccess to computing devices that can execute application programs.

FIG. 1B and FIG. 1C are block diagrams that depict embodiments ofcomputing devices.

FIG. 2A and FIG. 2B are block diagrams that depict embodiments of avirtualization environment.

FIG. 3 is a block diagram that depicts an embodiment of a system forremoting three-dimensional images using virtual machines.

FIGS. 4A-4B are flow diagrams that depict embodiments of methods forremoting three-dimensional images using virtual machines.

FIGS. 5A-5B are flow diagrams that depict embodiments of methods forallocating direct access to a graphics processing unit to a virtualmachine.

DETAILED DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates one embodiment of a computing environment 101 thatincludes one or more client machines 102A-102N (generally referred toherein as “client machine(s) 102”) that are in communication with one ormore servers 106A-106N (generally referred to herein as “server(s)106”). Installed in between the client machine(s) 102 and server(s) 106is a network.

In one embodiment, the computing environment 101 can include anappliance installed between the server(s) 106 and client machine(s) 102.This appliance can mange client/server connections, and in some casescan load balance client connections amongst a plurality of backendservers.

The client machine(s) 102 can in some embodiment be referred to as asingle client machine 102 or a single group of client machines 102,while server(s) 106 may be referred to as a single server 106 or asingle group of servers 106. In one embodiment a single client machine102 communicates with more than one server 106, while in anotherembodiment a single server 106 communicates with more than one clientmachine 102. In yet another embodiment, a single client machine 102communicates with a single server 106.

A client machine 102 can, in some embodiments, be referenced by any oneof the following terms: client machine(s) 102; client(s); clientcomputer(s); client device(s); client computing device(s); localmachine; remote machine; client node(s); endpoint(s); endpoint node(s);or a second machine. The server 106, in some embodiments, may bereferenced by any one of the following terms: server(s), local machine;remote machine; server farm(s), host computing device(s), or a firstmachine(s).

In one embodiment, the client machine 102 can be a virtual machine 102C.The virtual machine 102C can be any virtual machine, while in someembodiments the virtual machine 102C can be any virtual machine managedby a hypervisor developed by XenSolutions, Citrix Systems, IBM, VMware,or any other hypervisor. In other embodiments, the virtual machine 102Ccan be managed by any hypervisor, while in still other embodiments, thevirtual machine 102C can be managed by a hypervisor executing on aserver 106 or a hypervisor executing on a client 102.

The client machine 102 can in some embodiments execute, operate orotherwise provide an application that can be any one of the following:software; a program; executable instructions; a virtual machine; ahypervisor; a web browser; a web-based client; a client-serverapplication; a thin-client computing client; an ActiveX control; a Javaapplet; software related to voice over internet protocol (VoIP)communications like a soft IP telephone; an application for streamingvideo and/or audio; an application for facilitating real-time-datacommunications; a HTTP client; a FTP client; an Oscar client; a Telnetclient; or any other set of executable instructions. Still otherembodiments include a client device 102 that displays application outputgenerated by an application remotely executing on a server 106 or otherremotely located machine. In these embodiments, the client device 102can display the application output in an application window, a browser,or other output window. In one embodiment, the application is a desktop,while in other embodiments the application is an application thatgenerates a desktop.

The server 106, in some embodiments, executes a remote presentationclient or other client or program that uses a thin-client orremote-display protocol to capture display output generated by anapplication executing on a server 106 and transmits the applicationdisplay output to a remote client 102. The thin-client or remote-displayprotocol can be any one of the following protocols: the IndependentComputing Architecture (ICA) protocol manufactured by Citrix Systems,Inc. of Ft. Lauderdale, Fla.; or the Remote Desktop Protocol (RDP)manufactured by the Microsoft Corporation of Redmond, Wash.

The computing environment can include more than one server 106A-106Nsuch that the servers 106A-106N are logically grouped together into aserver farm 106. The server farm 106 can include servers 106 that aregeographically dispersed and logically grouped together in a server farm106, or servers 106 that are located proximate to each other andlogically grouped together in a server farm 106. Geographicallydispersed servers 106A-106N within a server farm 106 can, in someembodiments, communicate using a WAN, MAN, or LAN, where differentgeographic regions can be characterized as: different continents;different regions of a continent; different countries; different states;different cities; different campuses; different rooms; or anycombination of the preceding geographical locations. In some embodimentsthe server farm 106 may be administered as a single entity, while inother embodiments the server farm 106 can include multiple server farms106.

In some embodiments, a server farm 106 can include servers 106 thatexecute a substantially similar type of operating system platform (e.g.,WINDOWS NT, manufactured by Microsoft Corp. of Redmond, Wash., UNIX,LINUX, or SNOW LEOPARD.) In other embodiments, the server farm 106 caninclude a first group of servers 106 that execute a first type ofoperating system platform, and a second group of servers 106 thatexecute a second type of operating system platform. The server farm 106,in other embodiments, can include servers 106 that execute differenttypes of operating system platforms.

The server 106, in some embodiments, can be any server type. In otherembodiments, the server 106 can be any of the following server types: afile server; an application server; a web server; a proxy server; anappliance; a network appliance; a gateway; an application gateway; agateway server; a virtualization server; a deployment server; a SSL VPNserver; a firewall; a web server; an application server or as a masterapplication server; a server 106 executing an active directory; or aserver 106 executing an application acceleration program that providesfirewall functionality, application functionality, or load balancingfunctionality. In some embodiments, a server 106 may be a RADIUS serverthat includes a remote authentication dial-in user service. Inembodiments where the server 106 comprises an appliance, the server 106can be an appliance manufactured by any one of the followingmanufacturers: the Citrix Application Networking Group; Silver PeakSystems, Inc; Riverbed Technology, Inc.; F5 Networks, Inc.; or JuniperNetworks, Inc. Some embodiments include a first server 106A thatreceives requests from a client machine 102, forwards the request to asecond server 106B, and responds to the request generated by the clientmachine 102 with a response from the second server 106B. The firstserver 106A can acquire an enumeration of applications available to theclient machine 102 and well as address information associated with anapplication server 106 hosting an application identified within theenumeration of applications. The first server 106A can then present aresponse to the client's request using a web interface, and communicatedirectly with the client 102 to provide the client 102 with access to anidentified application.

The server 106 can, in some embodiments, execute any one of thefollowing applications: a thin-client application using a thin-clientprotocol to transmit application display data to a client; a remotedisplay presentation application; any portion of the CITRIX ACCESS SUITEby Citrix Systems, Inc. like the METAFRAME or CITRIX PRESENTATIONSERVER; MICROSOFT WINDOWS Terminal Services manufactured by theMicrosoft Corporation; or an ICA client, developed by Citrix Systems,Inc. Another embodiment includes a server 106 that is an applicationserver such as: an email server that provides email services such asMICROSOFT EXCHANGE manufactured by the Microsoft Corporation; a web orInternet server; a desktop sharing server; a collaboration server; orany other type of application server. Still other embodiments include aserver 106 that executes any one of the following types of hostedservers applications: GOTOMEETING provided by Citrix Online Division,Inc.; WEBEX provided by WebEx, Inc. of Santa Clara, Calif.; or MicrosoftOffice LIVE MEETING provided by Microsoft Corporation.

Client machines 102 can, in some embodiments, be a client node thatseeks access to resources provided by a server 106. In otherembodiments, the server 106 may provide clients 102 or client nodes withaccess to hosted resources. The server 106, in some embodiments,functions as a master node such that it communicates with one or moreclients 102 or servers 106. In some embodiments, the master node canidentify and provide address information associated with a server 106hosting a requested application, to one or more clients 102 or servers106. In still other embodiments, the master node can be a server farm106, a client 102, a cluster of client nodes 102, or an appliance.

One or more clients 102 and/or one or more servers 106 can transmit dataover a network 104 installed between machines and appliances within thecomputing environment 101. The network 104 can comprise one or moresub-networks, and can be installed between any combination of theclients 102, servers 106, computing machines and appliances includedwithin the computing environment 101. In some embodiments, the network104 can be: a local-area network (LAN); a metropolitan area network(MAN); a wide area network (WAN); a primary network 104 comprised ofmultiple sub-networks 104 located between the client machines 102 andthe servers 106; a primary public network 104 with a private sub-network104; a primary private network 104 with a public sub-network 104; or aprimary private network 104 with a private sub-network 104. Stillfurther embodiments include a network 104 that can be any of thefollowing network types: a point to point network; a broadcast network;a telecommunications network; a data communication network; a computernetwork; an ATM (Asynchronous Transfer Mode) network; a SONET(Synchronous Optical Network) network; a SDH (Synchronous DigitalHierarchy) network; a wireless network; a wireline network; or a network104 that includes a wireless link where the wireless link can be aninfrared channel or satellite band. The network topology of the network104 can differ within different embodiments, possible network topologiesinclude: a bus network topology; a star network topology; a ring networktopology; a repeater-based network topology; or a tiered-star networktopology. Additional embodiments may include a network 104 of mobiletelephone networks that use a protocol to communicate among mobiledevices, where the protocol can be any one of the following: AMPS; TDMA;CDMA; GSM; GPRS UMTS; or any other protocol able to transmit data amongmobile devices.

Illustrated in FIG. 1B is an embodiment of a computing device 100, wherethe client machine 102 and server 106 illustrated in FIG. 1A can bedeployed as and/or executed on any embodiment of the computing device100 illustrated and described herein. Included within the computingdevice 100 is a system bus 150 that communicates with the followingcomponents: a central processing unit 121; a main memory 122; storagememory 128; an input/output (I/O) controller 123; display devices124A-124N; an installation device 116; and a network interface 118. Inone embodiment, the storage memory 128 includes: an operating system,software routines, and a client agent 120. The I/O controller 123, insome embodiments, is further connected to a key board 126, and apointing device 127. Other embodiments may include an I/O controller 123connected to more than one input/output device 130A-130N.

FIG. 1C illustrates one embodiment of a computing device 100, where theclient machine 102 and server 106 illustrated in FIG. 1A can be deployedas and/or executed on any embodiment of the computing device 100illustrated and described herein. Included within the computing device100 is a system bus 150 that communicates with the following components:a bridge 170, and a first I/O device 130A. In another embodiment, thebridge 170 is in further communication with the main central processingunit 121, where the central processing unit 121 can further communicatewith a second I/O device 130B, a main memory 122, and a cache memory140. Included within the central processing unit 121, are I/O ports, amemory port 103, and a main processor.

Embodiments of the computing machine 100 can include a centralprocessing unit 121 characterized by any one of the following componentconfigurations: logic circuits that respond to and process instructionsfetched from the main memory unit 122; a microprocessor unit, such as:those manufactured by Intel Corporation; those manufactured by MotorolaCorporation; those manufactured by Transmeta Corporation of Santa Clara,Calif.; the RS/6000 processor such as those manufactured byInternational Business Machines; a processor such as those manufacturedby Advanced Micro Devices; or any other combination of logic circuits.Still other embodiments of the central processing unit 122 may includeany combination of the following: a microprocessor, a microcontroller, acentral processing unit with a single processing core, a centralprocessing unit with two processing cores, or a central processing unitwith more than one processing core.

While FIG. 1C illustrates a computing device 100 that includes a singlecentral processing unit 121, in some embodiments the computing device100 can include one or more processing units 121. In these embodiments,the computing device 100 may store and execute firmware or otherexecutable instructions that, when executed, direct the one or moreprocessing units 121 to simultaneously execute instructions or tosimultaneously execute instructions on a single piece of data. In otherembodiments, the computing device 100 may store and execute firmware orother executable instructions that, when executed, direct the one ormore processing units to each execute a section of a group ofinstructions. For example, each processing unit 121 may be instructed toexecute a portion of a program or a particular module within a program.

In some embodiments, the processing unit 121 can include one or moreprocessing cores. For example, the processing unit 121 may have twocores, four cores, eight cores, etc. In one embodiment, the processingunit 121 may comprise one or more parallel processing cores. Theprocessing cores of the processing unit 121, may in some embodimentsaccess available memory as a global address space, or in otherembodiments, memory within the computing device 100 can be segmented andassigned to a particular core within the processing unit 121. In oneembodiment, the one or more processing cores or processors in thecomputing device 100 can each access local memory. In still anotherembodiment, memory within the computing device 100 can be shared amongstone or more processors or processing cores, while other memory can beaccessed by particular processors or subsets of processors. Inembodiments where the computing device 100 includes more than oneprocessing unit, the multiple processing units can be included in asingle integrated circuit (IC). These multiple processors, in someembodiments, can be linked together by an internal high speed bus, whichmay be referred to as an element interconnect bus.

In embodiments where the computing device 100 includes one or moreprocessing units 121, or a processing unit 121 including one or moreprocessing cores, the processors can execute a single instructionsimultaneously on multiple pieces of data (SIMD), or in otherembodiments can execute multiple instructions simultaneously on multiplepieces of data (MIMD). In some embodiments, the computing device 100 caninclude any number of SIMD and MIMD processors.

The computing device 100, in some embodiments, can include a graphicsprocessor or a graphics processing unit (Not Shown). The graphicsprocessing unit can include any combination of software and hardware,and can further input graphics data and graphics instructions, render agraphic from the inputted data and instructions, and output the renderedgraphic. In some embodiments, the graphics processing unit can beincluded within the processing unit 121. In other embodiments, thecomputing device 100 can include one or more processing units 121, whereat least one processing unit 121 is dedicated to processing andrendering graphics.

One embodiment of the computing machine 100 includes a centralprocessing unit 121 that communicates with cache memory 140 via asecondary bus also known as a backside bus, while another embodiment ofthe computing machine 100 includes a central processing unit 121 thatcommunicates with cache memory via the system bus 150. The local systembus 150 can, in some embodiments, also be used by the central processingunit to communicate with more than one type of I/O device 130A-130N. Insome embodiments, the local system bus 150 can be any one of thefollowing types of buses: a VESA VL bus; an ISA bus; an EISA bus; aMicroChannel Architecture (MCA) bus; a PCI bus; a PCI-X bus; aPCI-Express bus; or a NuBus. Other embodiments of the computing machine100 include an I/O device 130A-130N that is a video display 124 thatcommunicates with the central processing unit 121. Still other versionsof the computing machine 100 include a processor 121 connected to an I/Odevice 130A-130N via any one of the following connections:HyperTransport, Rapid I/O, or InfiniBand. Further embodiments of thecomputing machine 100 include a processor 121 that communicates with oneI/O device 130A using a local interconnect bus and a second I/O device130B using a direct connection.

The computing device 100, in some embodiments, includes a main memoryunit 122 and cache memory 140. The cache memory 140 can be any memorytype, and in some embodiments can be any one of the following types ofmemory: SRAM; BSRAM; or EDRAM. Other embodiments include cache memory140 and a main memory unit 122 that can be any one of the followingtypes of memory: Static random access memory (SRAM), Burst SRAM orSynchBurst SRAM (BSRAM); Dynamic random access memory (DRAM); Fast PageMode DRAM (FPM DRAM); Enhanced DRAM (EDRAM), Extended Data Output RAM(EDO RAM); Extended Data Output DRAM (EDO DRAM); Burst Extended DataOutput DRAM (BEDO DRAM); Enhanced DRAM (EDRAM); synchronous DRAM(SDRAM); JEDEC SRAM; PC100 SDRAM; Double Data Rate SDRAM (DDR SDRAM);Enhanced SDRAM (ESDRAM); SyncLink DRAM (SLDRAM); Direct Rambus DRAM(DRDRAM); Ferroelectric RAM (FRAM); or any other type of memory. Furtherembodiments include a central processing unit 121 that can access themain memory 122 via: a system bus 150; a memory port 103; or any otherconnection, bus or port that allows the processor 121 to access memory122.

One embodiment of the computing device 100 provides support for any oneof the following installation devices 116: a CD-ROM drive, a CD-R/RWdrive, a DVD-ROM drive, tape drives of various formats, USB device, abootable medium, a bootable CD, a bootable CD for GNU/Linux distributionsuch as KNOPPIX®, a hard-drive or any other device suitable forinstalling applications or software. Applications can in someembodiments include a client agent 120, or any portion of a client agent120. The computing device 100 may further include a storage device 128that can be either one or more hard disk drives, or one or moreredundant arrays of independent disks; where the storage device isconfigured to store an operating system, software, programsapplications, or at least a portion of the client agent 120. A furtherembodiment of the computing device 100 includes an installation device116 that is used as the storage device 128.

The computing device 100 may further include a network interface 118 tointerface to a Local Area Network (LAN), Wide Area Network (WAN) or theInternet through a variety of connections including, but not limited to,standard telephone lines, LAN or WAN links (e.g., 802.11, T1, T3, 56 kb,X.25, SNA, DECNET), broadband connections (e.g., ISDN, Frame Relay, ATM,Gigabit Ethernet, Ethernet-over-SONET), wireless connections, or somecombination of any or all of the above. Connections can also beestablished using a variety of communication protocols (e.g., TCP/IP,IPX, SPX, NetBIOS, Ethernet, ARCNET, SONET, SDH, Fiber Distributed DataInterface (FDDI), RS232, RS485, IEEE 802.11, IEEE 802.11a, IEEE 802.11b,IEEE 802.11g, CDMA, GSM, WiMax and direct asynchronous connections). Oneversion of the computing device 100 includes a network interface 118able to communicate with additional computing devices 100′ via any typeand/or form of gateway or tunneling protocol such as Secure Socket Layer(SSL) or Transport Layer Security (TLS), or the Citrix Gateway Protocolmanufactured by Citrix Systems, Inc. Versions of the network interface118 can comprise any one of: a built-in network adapter; a networkinterface card; a PCMCIA network card; a card bus network adapter; awireless network adapter; a USB network adapter; a modem; or any otherdevice suitable for interfacing the computing device 100 to a networkcapable of communicating and performing the methods and systemsdescribed herein.

Embodiments of the computing device 100 include any one of the followingI/O devices 130A-130N: a keyboard 126; a pointing device 127; mice;trackpads; an optical pen; trackballs; microphones; drawing tablets;video displays; speakers; inkjet printers; laser printers; anddye-sublimation printers; or any other input/output device able toperform the methods and systems described herein. An I/O controller 123may in some embodiments connect to multiple I/O devices 103A-130N tocontrol the one or more I/O devices. Some embodiments of the I/O devices130A-130N may be configured to provide storage or an installation medium116, while others may provide a universal serial bus (USB) interface forreceiving USB storage devices such as the USB Flash Drive line ofdevices manufactured by Twintech Industry, Inc. Still other embodimentsinclude an I/O device 130 that may be a bridge between the system bus150 and an external communication bus, such as: a USB bus; an AppleDesktop Bus; an RS-232 serial connection; a SCSI bus; a FireWire bus; aFireWire 800 bus; an Ethernet bus; an AppleTalk bus; a Gigabit Ethernetbus; an Asynchronous Transfer Mode bus; a HIPPI bus; a Super HIPPI bus;a SerialPlus bus; a SCI/LAMP bus; a FibreChannel bus; or a SerialAttached small computer system interface bus.

In some embodiments, the computing machine 100 can connect to multipledisplay devices 124A-124N, in other embodiments the computing device 100can connect to a single display device 124, while in still otherembodiments the computing device 100 connects to display devices124A-124N that are the same type or form of display, or to displaydevices that are different types or forms. Embodiments of the displaydevices 124A-124N can be supported and enabled by the following: one ormultiple I/O devices 130A-130N; the I/O controller 123; a combination ofI/O device(s) 130A-130N and the I/O controller 123; any combination ofhardware and software able to support a display device 124A-124N; anytype and/or form of video adapter, video card, driver, and/or library tointerface, communicate, connect or otherwise use the display devices124A-124N. The computing device 100 may in some embodiments beconfigured to use one or multiple display devices 124A-124N, theseconfigurations include: having multiple connectors to interface tomultiple display devices 124A-124N; having multiple video adapters, witheach video adapter connected to one or more of the display devices124A-124N; having an operating system configured to support multipledisplays 124A-124N; using circuits and software included within thecomputing device 100 to connect to and use multiple display devices124A-124N; and executing software on the main computing device 100 andmultiple secondary computing devices to enable the main computing device100 to use a secondary computing device's display as a display device124A-124N for the main computing device 100. Still other embodiments ofthe computing device 100 may include multiple display devices 124A-124Nprovided by multiple secondary computing devices and connected to themain computing device 100 via a network.

In some embodiments, the computing machine 100 can execute any operatingsystem, while in other embodiments the computing machine 100 can executeany of the following operating systems: versions of the MICROSOFTWINDOWS operating systems such as WINDOWS 3.x; WINDOWS 95; WINDOWS 98;WINDOWS 2000; WINDOWS NT 3.51; WINDOWS NT 4.0; WINDOWS CE; WINDOWS XP;and WINDOWS VISTA; the different releases of the Unix and Linuxoperating systems; any version of the MAC OS manufactured by AppleComputer; OS/2, manufactured by International Business Machines; anyembedded operating system; any real-time operating system; any opensource operating system; any proprietary operating system; any operatingsystems for mobile computing devices; or any other operating system. Instill another embodiment, the computing machine 100 can execute multipleoperating systems. For example, the computing machine 100 can executePARALLELS or another virtualization platform that can execute or managea virtual machine executing a first operating system, while thecomputing machine 100 executes a second operating system different fromthe first operating system.

The computing machine 100 can be embodied in any one of the followingcomputing devices: a computing workstation; a desktop computer; a laptopor notebook computer; a server; a handheld computer; a mobile telephone;a portable telecommunication device; a media playing device; a gamingsystem; a mobile computing device; a netbook; a device of the IPODfamily of devices manufactured by Apple Computer; any one of thePLAYSTATION family of devices manufactured by the Sony Corporation; anyone of the Nintendo family of devices manufactured by Nintendo Co; anyone of the XBOX family of devices manufactured by the MicrosoftCorporation; or any other type and/or form of computing,telecommunications or media device that is capable of communication andthat has sufficient processor power and memory capacity to perform themethods and systems described herein. In other embodiments the computingmachine 100 can be a mobile device such as any one of the followingmobile devices: a JAVA-enabled cellular telephone or personal digitalassistant (PDA), such as the i55sr, i58sr, i85s, i88s, i90c, i95cl, orthe im1100, all of which are manufactured by Motorola Corp; the 6035 orthe 7135, manufactured by Kyocera; the i300 or i330, manufactured bySamsung Electronics Co., Ltd; the TREO 180, 270, 600, 650, 680, 700p,700w, or 750 smart phone manufactured by Palm, Inc; any computing devicethat has different processors, operating systems, and input devicesconsistent with the device; or any other mobile computing device capableof performing the methods and systems described herein. In still otherembodiments, the computing device 100 can be any one of the followingmobile computing devices: any one series of Blackberry, or otherhandheld device manufactured by Research In Motion Limited; the iPhonemanufactured by Apple Computer; Palm Pre; a Pocket PC; a Pocket PCPhone; or any other handheld mobile device.

Illustrated in FIG. 2A is one embodiment of a virtualizationenvironment. Included on a computing device 201 is a hardware layer thatcan include one or more physical disks 204, one or more physical devices206, one or more physical processors 208 and a physical memory 216. Insome embodiments, firmware 212 can be stored within a memory element inthe physical memory 216 and can be executed by one or more of thephysical processors 208. The computing device 201 can further include anoperating system 214 that can be stored in a memory element in thephysical memory 216 and executed by one or more of the physicalprocessors 208. Still further, a hypervisor 202 can be stored in amemory element in the physical memory 216 and can be executed by one ormore of the physical processors 208. Executing on one or more of thephysical processors 208 can be one or more virtual machines 232A-C(generally 232). Each virtual machine 232 can have a virtual disk 226A-Cand a virtual processor 228A-C. In some embodiments, a first virtualmachine 232A can execute, on a virtual processor 228A, a control program220 that includes a tools stack 224. In other embodiments, one or morevirtual machines 232B-C can executed, on a virtual processor 228B-C, aguest operating system 230A-B.

Further referring to FIG. 2A, and in more detail, in one embodiment thevirtualization environment described includes a Type 2 hypervisor 202,or a hypervisor that executes within an operating system 214 executingon the computing device 201. A Type 2 hypervisor, in some embodiments,executes within an operating system 214 environment and virtual machinesexecute at a level above the hypervisor. In many embodiments, the Type 2hypervisor executes within the context of a user's operating system suchthat the Type 2 hypervisor interacts with the user's operating system.

In some embodiments, the virtualization environment includes a computingdevice 201. The computing device 201 can be any computing device, and insome embodiments the computing device 201 can be any computer, device orcomputing machine described herein. While FIG. 2A illustrates a singlecomputing device 201, in some embodiments the modules, programs, virtualmachines, and commands stored and executed by the computing device 201can be executed by more than one computing device 201. In still otherembodiments, the computing device 201 can be a server farm.

In one embodiment, the computing device 201 can include a hardware layer210 that includes one or more pieces of hardware that communicates withthe computing machine 201. In some embodiments, the hardware layer 210can include any hardware included in the computing device 201. In otherembodiments, the hardware layer 210 can include one or more physicaldisks 204, one or more physical devices 206, one or more physicalprocessors 208 and memory 216.

The hardware layer 210, in some embodiments, can include one or morephysical disks 204. A physical disk 204 can be any hard disk, while insome embodiments a physical disk 204 can be any hard disk describedherein. In some embodiments, the hardware layer 210 can include onephysical disk 204. In other embodiments, the hardware layer 210 caninclude more than one physical disk 204. The computing device 201, insome embodiments, can communicate with an external hard disk that isincluded in the hardware layer 210 as a physical disk 204.

In other embodiments, the hardware layer 210 can include a processor208. The processor 208, in some embodiments, can be any processor, whilein other embodiments the processor 208 can be any processor describedherein. The processor 208 can include one or more processing cores. Inother embodiments the computing device 201 can include one or moreprocessors 208. In some embodiments, the computing device 201 caninclude one or more different processors, e.g. a processing unit, agraphics processing unit, or a physics engine.

Physical devices 206, in some embodiments, can be any device included inthe computing device 201. In some embodiments, physical devices 206 canbe any combination of devices included in the computing device 201 andexternal devices that communicate with the computing device 201. Thecomputing device 201, in some embodiments, can include one or morephysical devices 206. A physical device 206 can be any of the following:a network interface card; a video card; a keyboard; a mouse; an inputdevice; a monitor; a display device; speakers; an optical drive; astorage device; a universal serial bus connection; any device connectedto the computing device 201; any device communicating with the computingdevice 201; a printer; a scanner; or any other device or devicedescribed herein.

The hardware layer 210 can further include physical memory 216 that caninclude any type of memory. In some embodiments, the physical memory 216can include any memory type described herein. The physical memory 216can store data, and in some embodiments can store one or more programs,or set of executable instructions. FIG. 2A illustrates one embodimentwhere firmware 212 is stored within the physical memory 216 of thecomputing device 201. Programs or executable instructions stored in thephysical memory 216 can be executed by the one or more processors 208 ofthe computing device 201.

Firmware 212, in some embodiments, can be any combination of executableinstructions and hardware that controls hardware communicating with orincluded within the computing device 201. In some embodiments, thefirmware 212 can control one or more pieces of hardware within thehardware layer 210. Firmware 212, in many embodiments, can be executedby one or more processors 208 within the computing device 201. In someembodiments, the firmware 212 can be boot firmware such as the basicinput/output system (BIOS.) Additional firmware 212 executing on thecomputing device 201 can interface with the BIOS.

In one embodiment, the computing device 201 can include an operatingsystem 214 executed by one or more physical processors 208. In someembodiments, the operating system 214 is a user operating system thatcan directly access the hardware devices in the hardware layer 210. Theoperating system 214 can be any operating system and in someembodiments, the operating system 214 can be any operating systemdescribed herein. FIG. 2A illustrates one embodiment where thehypervisor 202 executes within the context of the operating system 214executing on the computing device 201. In this embodiment, the operatingsystem 214 can be referred to as a host operating system 214, while theother operating systems can be referred to as guest operating systems.Guest operating systems can include the guest operating systems 230A-Bexecuting on the virtual machines 232, and/or the control program 220.

In some embodiments, the computing device 201 can include a hypervisor202. A hypervisor 202, in some embodiments, can be a program thatexecuted by processors 208 on the computing device 201 to manage anynumber of virtual machines. The hypervisor 202 can be referred to as avirtual machine monitor, or platform virtualization software. In someembodiments, a hypervisor 202 can be any combination of executableinstructions and hardware that monitors virtual machines executing on acomputing machine. While FIG. 2A illustrates a virtualizationenvironment that includes a Type 2 hypervisor 202, the computing device201 can execute any other type of hypervisor. For example, the computingdevice 201 can execute a virtualization environment that includes a Type1 hypervisor 202. In some embodiments, the computing device 201 canexecute one or more hypervisors 202. These one or more hypervisors 202can be the same type of hypervisor, or in other embodiments can bedifferent hypervisor types.

The hypervisor 202, in some embodiments, can provide virtual resourcesto operating systems 230 or control programs 220 executing on virtualmachines 232 in any manner that simulates the operating systems 230 orcontrol programs 220 having direct access to system resources. Systemresources can include: physical devices; physical disks; physicalprocessors; physical memory 216 and any other component included in thecomputing device 201 hardware layer 210. In these embodiments, thehypervisor 202 may be used to emulate virtual hardware, partitionphysical hardware, virtualize physical hardware, or execute virtualmachines that provide access to computing environments. In still otherembodiments, the hypervisor 202 controls processor scheduling and memorypartitioning for a virtual machine 232 executing on the computing device201. Hypervisor 202 may include those manufactured by VMWare, Inc., ofPalo Alto, Calif.; the XEN hypervisor, an open source product whosedevelopment is overseen by the open source Xen.org community; HyperV,VirtualServer or virtual PC hypervisors provided by Microsoft, orothers. In some embodiments, a computing device 201 executes ahypervisor 202 that creates a virtual machine platform on which guestoperating systems may execute. In these embodiments, the computingdevice 201 can be referred to as a host server. An example of such acomputing device is the XEN SERVER provided by Citrix Systems, Inc., ofFort Lauderdale, Fla.

In one embodiment, the hypervisor 202 can create a virtual machine232A-B (generally 232) in which an operating system 230 executes. In oneof these embodiments, for example, the hypervisor 202 loads a virtualmachine image to create a virtual machine 232. In another of theseembodiments, the hypervisor 202 executes an operating system 230 withinthe virtual machine 232. In still another of these embodiments, thevirtual machine 232 executes an operating system 230.

In one embodiment, the hypervisor 202 controls the execution of at leastone virtual machine 232. In another embodiment, the hypervisor 202presents at least one virtual machine 232 with an abstraction of atleast one hardware resource provided by the computing device 201. Theabstraction can further be referred to as a virtualization or virtualview of the hardware, memory processor and other system resourcesavailable on the computing device 201. Hardware or hardware resources,in some embodiments, can be any hardware resource available within thehardware layer 210. In other embodiments, the hypervisor 202 controlsthe manner in which virtual machines 232 access the physical processors208 available in the computing device 201. Controlling access to thephysical processors 208 can include determining whether a virtualmachine 232 should have access to a processor 208, and how physicalprocessor capabilities are presented to the virtual machine 232.

In some embodiments, the computing device 201 can host or execute one ormore virtual machines 232. A virtual machine 232 can be called a domain,a guest and/or a DOMAIN U. A virtual machine 232 is a set of executableinstructions that, when executed by a processor 208, imitate theoperation of a physical computer such that the virtual machine 232 canexecute programs and processes much like a physical computing device.While FIG. 2A illustrates an embodiment where a computing device 201hosts three virtual machines 232, in other embodiments the computingdevice 201 can host any number of virtual machines 232. The hypervisor202, in some embodiments, provides each virtual machine 232 with aunique virtual view of the physical hardware, memory, processor andother system resources available to that virtual machine 232. In someembodiments, the unique virtual view can be based on any of thefollowing: virtual machine permissions; application of a policy engineto one or more virtual machine identifiers; the user accessing a virtualmachine; the applications executing on a virtual machine; networksaccessed by a virtual machine; or any other similar criteria. Thehypervisor 202, in other embodiments, provides each virtual machine 232with a substantially similar virtual view of the physical hardware,memory, processor and other system resources available to the virtualmachines 232.

Each virtual machine 232 can include a virtual disk 226A-C (generally226) and a virtual processor 228A-C (generally 228.) The virtual disk226, in some embodiments, is a virtualized view of one or more physicaldisks 204 of the computing device 201, or a portion of one or morephysical disks 204 of the computing device 201. The virtualized view ofthe physical disks 204 can be generated, provided and managed by thehypervisor 202. In some embodiments, the hypervisor 202 provides eachvirtual machine 232 with a unique view of the physical disks 204. Thus,in these embodiments, the virtual disk 226 included in each virtualmachine 232 can be unique when compared with the other virtual disks226.

A virtual processor 228 can be a virtualized view of one or morephysical processors 208 of the computing device 201. In someembodiments, the virtualized view of the physical processors 208 can begenerated, provided and managed by the hypervisor 202. In someembodiments, the virtual processor 228 has substantially all of the samecharacteristics of at least one physical processor 208. In otherembodiments, the virtual processor 208 provides a modified view of thephysical processors 208 such that at least some of the characteristicsof the virtual processor 228 are different than the characteristics ofthe corresponding physical processor 208.

A control program 220 may execute at least one application for managingand configuring the guest operating systems executing on the virtualmachines 232 and in some embodiments the computing device 201. In someembodiments, the control program 220 can be called a control operatingsystem, a control domain, domain 0 or dom 0. The control program 220, insome embodiments, can be DOMAIN o or DOM0 of the XEN hypervisor. Thecontrol program 220 can execute an administrative application or programthat can further display a user interface which administrators can useto access the functionality of each virtual machine 232 and/or to managethe virtual machines 232. In some embodiments, the user interfacegenerated by the administrative program can be used to terminate theexecution of virtual machines 232, allocate resources to virtualmachines 232, assign permissions to virtual machines 232, or managesecurity credentials associated with virtual machines 232. The controlprogram 220, in some embodiments, can start new virtual machines 232 orterminate execution of executing virtual machines 232. In otherembodiments, the control program 220 can directly access hardware and/orresources within the hardware layer 210. In still another embodiment,the control program 220 can interface with programs and applicationsexecuting on the computing device 210 and outside of the context of avirtual machine 232. Similarly, the control program 220 can interfacewith programs and applications executing within the context of a virtualmachine 232.

In one embodiment, the hypervisor 202 can execute the control program220 within a virtual machine 232. The hypervisor 202 can create andstart the virtual machine 232. In embodiments where the hypervisor 202executes the control program 220 within a virtual machine 232, thatvirtual machine 232 can be referred to as the control virtual machine232. In still another embodiment, the control program 220 executeswithin a virtual machine 232 that is authorized to directly accessphysical resources on the computing device 201.

In some embodiments, a control program 220A (Not Shown) on a firstcomputing device 201A (Not Shown) may exchange data with a controlprogram 220B (Not Shown) on a second computing device 201B (Not Shown).In these embodiments the first computing device 201A may be locatedremote from the second computing device 201B. The control programs220A-B can exchange data via a communication link between a hypervisor202A (Not Shown) executing on the first computing device 201A and ahypervisor 202B (Not Shown) executing on the second computing device201B. Through this communication link, the computing devices 201A-B canexchange data regarding processors and other physical resourcesavailable in a pool of resources. Further, through this connectionbetween hypervisors 202A-B, the hypervisors 202A-B can manage a pool ofresources, e.g. the resources available on the first computing device201A and the second computing device 201B, distributed across one ormore computing devices 201A-B. The hypervisors 202A-B can furthervirtualize these resources and make them available to virtual machines232 executing on the computing devices 201A-B. In another instance ofthis embodiment, a single hypervisor 202 can manage and control virtualmachines 232 executing on both computing devices 201A-B.

In some embodiments, the control program 220 interacts with one or moreguest operating systems 230A-B (generally 230.) The control program 220can communicate with the guest operating systems 230 through ahypervisor 202. Through the hypervisor 202, the guest operating system230 can request access to physical disks 204, physical processors 208,memory 216, physical devices 206 and any other component in the hardwarelayer 210. In still other embodiments, the guest operating systems 230can communicate with the control program 220 via a communication channelestablished by the hypervisor 202, such as, for example, via a pluralityof shared memory pages made available by the hypervisor 202.

In some embodiments, the control program 220 includes a network back-enddriver for communicating directly with networking hardware provided bythe computing device 201. In one of these embodiments, the networkback-end driver processes at least one virtual machine request from atleast one guest operating system 230. In other embodiments, the controlprogram 220 includes a block back-end driver for communicating with astorage element on the computing device 201. In one of theseembodiments, the block back-end driver reads and writes data from thestorage element based upon at least one request received from a guestoperating system 230.

In another embodiment, the control program 220 includes a tools stack224. In another embodiment, a tools stack 224 provides functionality forinteracting with the hypervisor 202, communicating with other controlprograms 220 (for example, on a second computing device 201B), ormanaging virtual machines 232 on the computing device 201. In anotherembodiment, the tools stack 224 includes customized applications forproviding improved management functionality to an administrator of avirtual machine farm. In some embodiments, at least one of the toolsstack 224 and the control program 220 include a management API thatprovides an interface for remotely configuring and controlling virtualmachines 232 running on a computing device 201. In other embodiments,the control program 220 communicates with the hypervisor 202 through thetools stack 224.

In one embodiment, the hypervisor 202 executes a guest operating system230 within a virtual machine 232 created by the hypervisor 202. Inanother embodiment, the guest operating system 230 provides a user ofthe computing device 201 with access to resources within a computingenvironment. In still another embodiment, a resource includes a program,an application, a document, a file, a plurality of applications, aplurality of files, an executable program file, a desktop environment, acomputing environment, or other resource made available to a user of thecomputing device 201. In yet another embodiment, the resource may bedelivered to the computing device 201 via a plurality of access methodsincluding, but not limited to, conventional installation directly on thecomputing device 201, delivery to the computing device 201 via a methodfor application streaming, delivery to the computing device 201 ofoutput data generated by an execution of the resource on a secondcomputing device 201′ and communicated to the computing device 201 via apresentation layer protocol, delivery to the computing device 201 ofoutput data generated by an execution of the resource via a virtualmachine executing on a second computing device 201′, or execution from aremovable storage device connected to the computing device 201, such asa USB device, or via a virtual machine executing on the computing device201 and generating output data. In some embodiments, the computingdevice 201 transmits output data generated by the execution of theresource to another computing device 201′.

In one embodiment, the guest operating system 230, in conjunction withthe virtual machine on which it executes, forms a fully-virtualizedvirtual machine that is not aware that it is a virtual machine; such amachine may be referred to as a “Domain U HVM (Hardware Virtual Machine)virtual machine”. In another embodiment, a fully-virtualized machineincludes software emulating a Basic Input/Output System (BIOS) in orderto execute an operating system within the fully-virtualized machine. Instill another embodiment, a fully-virtualized machine may include adriver that provides functionality by communicating with the hypervisor202. In such an embodiment, the driver is typically aware that itexecutes within a virtualized environment.

In another embodiment, the guest operating system 230, in conjunctionwith the virtual machine on which it executes, forms a paravirtualizedvirtual machine, which is aware that it is a virtual machine; such amachine may be referred to as a “Domain U PV virtual machine”. Inanother embodiment, a paravirtualized machine includes additionaldrivers that a fully-virtualized machine does not include. In stillanother embodiment, the paravirtualized machine includes the networkback-end driver and the block back-end driver included in a controlprogram 220, as described above.

Illustrated in FIG. 2B is another embodiment of a virtualizationenvironment that illustrates a Type 1 hypervisor 202. Executing on thecomputing device 201 is a hypervisor 202 that can directly access thehardware and resources within the hardware layer 210. Virtual machines232 managed by the hypervisor 202 can be an unsecure virtual machine232B and/or a secure virtual machine 232C. Whereas the virtualizationenvironment depicted in FIG. 2A illustrates a host operating system 214,the virtualization environment embodiment in FIG. 2B does not execute ahost operating system.

Further referring to FIG. 2B, and in more detail, the virtualizationenvironment includes a Type 1 hypervisor 202. Type 1 hypervisors 202, insome embodiments, execute on “bare metal,” such that the hypervisor 202has direct access to all applications and processes executing on thecomputing device 201, all resources on the computing device 201 and allhardware on the computing device 201 or communicating with the computingdevice 201. While a Type 2 hypervisor 202 accesses system resourcesthrough a host operating system 214, a Type 1 hypervisor 202 candirectly access all system resources. The Type 1 hypervisor 202 canexecute directly on one or more physical processors of the computingdevice 201, and can include program data stored in the physical memory216.

In a virtualization environment that employs a Type 1 hypervisor 202configuration, the host operating system can be executed by one or morevirtual machines 232. Thus, a user of the computing device 201 candesignate one or more virtual machines 232 as the user's personalmachine. This virtual machine can imitate the host operating system byallowing a user to interact with the computing device 201 insubstantially the same manner that the user would interact with thecomputing device 201 via a host operating system 214.

Virtual machines 232 can be unsecure virtual machines 232B and securevirtual machine 232C. While FIG. 2B illustrates a secure and unsecurevirtual machine, sometimes they can be referred to as privileged andunprivileged virtual machines. In some embodiments, a virtual machine'ssecurity can be determined based on a comparison of the virtual machineto other virtual machines executing within the same virtualizationenvironment. For example, were a first virtual machine to have access toa pool of resources, and a second virtual machine not to have access tothe same pool of resources; the second virtual machine could beconsidered an unsecure virtual machine 232B while the first virtualmachine could be considered a secure virtual machine 232A. In someembodiments, a virtual machine's 323 ability to access one or moresystem resources can be configured using a configuration interfacegenerated by either the control program 220 or the hypervisor 202. Inother embodiments, the level of access afforded to a virtual machine 232can be the result of a review of any of the following sets of criteria:the user accessing the virtual machine; one or more applicationsexecuting on the virtual machine; the virtual machine identifier; a risklevel assigned to the virtual machine based on one or more factors; orany other similar criteria.

In some embodiments, unsecure virtual machines 232B may be preventedfrom accessing resources, hardware, memory locations, and programs thatsecure virtual machines 232A may access. For example, a secure virtualmachine 232C may be able to access one or more company resources, whilethe unsecure virtual machine 232B cannot access any company resources.

Illustrated in FIG. 3 is one embodiment of a system for remotelyrendering three-dimensional images using virtual machines. The systemcan include a computing device 201 having a hardware layer 210 that caninclude a graphics processing unit (GPU) 305. The computing device 201can execute a virtualization environment 302 that includes a hypervisor202 and one or more virtual machines 232A-232D (herein referred togenerally as virtual machines 232). In some instances, one of thevirtual machines can be a control virtual machine 232A that can executea control program 220 and can include a virtual disk 226A and a virtualprocessor 228A. Additional virtual machines 232B-232D can execute withinthe virtualization environment 302. These virtual machines 232B-232D canexecute a guest operating system 230A-230C (referred to generally as aguest operating system 230), a rendering agent 315A-315C (referred togenerally as a rendering agent 315), a graphics driver 320A-320C(referred to generally as a graphics driver 320), a remoting agent325A-325C (referred to generally as a remoting agent 325), or any numberof three-dimensional applications 330A-330C (referred to generally as athree-dimensional application 330). These virtual machines 232B-232D canalso include a virtual disk 226 and a virtual processor 228. Thecomputing device 201 can communicate with any number of additionalcomputing devices 340A-340B. Each of these computing devices 340A-340Bcan execute a remoting application 335A-335B (referred to generally as aremoting application 335).

Referring to FIG. 3, and in more detail, in one embodiment the computingdevice 201 and the remote or other computing devices 340A-340B can beany computer, device or computing device described herein. In someembodiments, the computing device 201 and the other computing devices340A-340B can be referred to as a computer or a computing machine. Theother computing devices 340A-340B can be generally referred to as remotecomputing devices 340, as they are located in a remote location from thecomputing device 201. In some embodiments, the remote computing devices340 can be referred to as client computing devices while the computingdevice 201 can be referred to as a server. The computing device 201, andthe remote computing devices 340 can in some embodiments be physicalcomputers. While FIG. 3 illustrates two remote computing devices 340,the computing device 201 can communicate with any number of remotecomputing devices 340. The remote computing devices 340 can be referredto as first, second or third computing devices.

The computing device 201 can communicate with remote computing devices340 over a network such as any network described herein. In someembodiments, communication between the computing device 201 and theremote computing devices 340 can occur over a communication or virtualchannel established between the computing device 201 and the remotecomputing devices 340. This communication channel can include any numberof sub-channels, and in some embodiments the communication channel canbe established using a remote display protocol such as RDP or the CITRIXICA protocol. In one embodiment, the communication channel can beestablished between a remoting agent 325 on the computing device 201 anda remoting application 335 executing on a remote device 340.

The remoting application 335, in some embodiments, can be an applicationthat can establish one or more communication channels with a remotecomputer 340. In some embodiments, the remoting application 335 cantransmit application output to a remote computer for display, or canreceive application output generated by an application executing on aremote computer. When the remoting application 335 receives applicationoutput generated by a remote application, the remoting application candisplay the received output in an application output window or adesktop.

In some instances, the remoting agent 325 can function substantially thesame as the remoting application 335. The remoting agent 325 can receiveapplication output generated by a locally executing application and cantransmit that application output to a remote computer 340. In someembodiments, the remoting agent 325 can transmit the application outputto a remoting application 335 executing on a remote computer 340. Theremoting agent 325 can communicate with any of the applications orobjects executing within the virtualization environment 302. In oneembodiment, the remoting agent 325 can receive or obtainthree-dimensional images from a graphics driver 320, a three-dimensionalapplication 330, a rendering agent 315, storage buffer, or any otherapplication, storage repository or object. Upon receiving or obtainingthe three-dimensional images, the remoting agent 325 can transmit thereceived images to a remote computer 340. In some embodiments, theremoting agent 325 can receive three-dimensional draw commands insteadof three-dimensional images. In those embodiments, the remoting agent325 can transmit the received draw commands to a remote computer 340.

The hardware layer 210, in some embodiments, can be any hardware layer210 described herein and can include any computer hardware describedherein. FIG. 3 illustrates a hardware layer 210 that includes a graphicsprocessing unit (GPU) 305. The GPU 305 can be a software applicationthat receives three-dimensional draw commands and uses a graphicslibrary to render three-dimensional images from the three-dimensionaldraw commands. In other embodiments, the GPU 305 can be a hardwareelement that receives three-dimensional draw commands and uses agraphics library to render three-dimensional images from thethree-dimensional draw commands. In some embodiments the GPU 305 can bea combination of software or hardware, while in other embodiments theGPU 305 can be included in a central processing unit. In still otherembodiments, the GPU 305 can reside on a graphics card or can compriseany number of GPUs, where each GPU may be dedicated to rendering aparticular image aspect, e.g. texture, shading, etc. The GPU 305 canrender both two-dimensional and three-dimensional images and can use anygraphics library. In one embodiment, the GPU 305 can renderthree-dimensional images using any of the following graphics libraries:DIRECT3D; OGRE; OPENGL; QUESA; TEDDY3D; MESA3D; or any other graphicslibrary that can be used to render three-dimensional images.

The virtualization environment 302, in some embodiments, can be anyvirtualization environment 302. In other embodiments, the virtualizationenvironment 302 can be any virtualization environment 302 describedherein. The virtualization environment 302 can contain the componentsnecessary to generate, manage and execute at least one virtual machineon the physical computer 201. In some embodiments, the virtualizationenvironment 302 can include a hypervisor 202. The hypervisor 202 cancommunicate with the hardware layer 210 of the physical computer 201 andin some embodiments can virtualize the hardware layer 210. In manyembodiments, the hypervisor 202 can communicate with each of the virtualmachines 232 and the control program 232. The hypervisor 202 can be anyhypervisor and in some embodiments can be any hypervisor describedherein. In some embodiments the hypervisor 202 can execute directly onthe hardware of the computer 201, while in other embodiments thehypervisor 202 can execute on top of an operating system. The hypervisor202, in some embodiments, can be the XEN hypervisor published by CITRIXSYSTEMS.

In some embodiments, a control virtual machine 232A can execute withinthe virtualization environment 302. The control virtual machine 232A canbe any control virtual machine 232A described herein. In one embodiment,the control virtual machine 232A can be referred to as the Domain 0virtual machine. In some embodiments, the control virtual machine 232Acan execute a control program 220 such as any control program describedherein. In still other embodiments, the control virtual machine 232A caninclude a virtual processor 228 and a virtual disk 226 such as anyvirtual disk or virtual processor described herein. While FIG. 3illustrates a control virtual machine 232A that includes a controlprogram 220, a virtual disk 226 and a virtual processor 228, in otherembodiments the control virtual machine 232A can include any of thecomponents included in the other virtual machines 232B-232D, e.g. guestoperating system 230, rendering agent 315, remoting agent 325,three-dimensional applications 330, and a graphics driver 320.

The additional virtual machines 232 can be any virtual machine 232described herein. In some embodiments, the additional virtual machines232B-232D can be managed by the control virtual machine 232A and/or thecontrol program 220. The additional virtual machines 232 can include avirtual disk 226B-226D such as any virtual disk described herein, andcan include a virtual processor 228B-228D such as any virtual processordescribed herein. In some embodiments, the additional virtual machines232 can execute a guest operating system 230 such as any guest operatingsystem described herein.

The guest operating system 230 executed by a virtual machine 232 can insome embodiments be a paravirtualized operating system. Paravirtualizedoperating systems can be operating systems that are configured tooperate within the context of a virtual environment. In manyembodiments, paravirtualized operating systems provide applications,objects and components executing within the context of theparavirtualized operating system, direct access to the physical hardwarevirtualized by the hypervisor 202. In other embodiments, the guestoperating system 230 can be a non-paravirtualized operating system.Non-paravirtualized operating systems can be operating systems that arenot configured to operate within the context of a virtual environment.In many embodiments, non-paravirtualized operating systems provideapplications, objects and components executing within the context of thenon-paravirtualized operating system, direct access to the physicalhardware virtualized by the hypervisor 202. The physical hardware, insome embodiments, can be any hardware included in the hardware layer210. In other embodiments, the physical hardware can be any hardware incommunication with the hypervisor 202 and virtualized by the hypervisor202.

In some embodiments, each virtual machine 232 can include a graphicsdriver 320. The graphics driver 320 can be any graphics driver 320 andin some embodiment can be a three-dimensional graphics driver 320 ableto intercept three-dimensional draw commands and forward the commands toa GPU 305 for rendering. In some embodiments, the graphics driver 320can be a graphics driver 320 compatible with any GPU 305, and a graphicsdriver that can intercept and forward for rendering three-dimensionaldraw commands interpretable by any of the following graphics libraries:RECT3D; OGRE; OPENGL; QUESA; TEDDY3D; MESA3D; or any other graphicslibrary that can be used to render two or three-dimensional images. Insome embodiments, the graphics driver 320 can be an agent. For example,in one embodiment the graphics driver 320, i.e. the agent, can executeon one virtual machine 232C and can intercept three-dimensional drawcommands generated by a three-dimensional application executing on theremote computer 340. Upon intercepting the three-dimensional drawcommands, the agent 320 can forward the three-dimensional draw commandsto a rendering agent 315 executing on another virtual machine 232B thathas direct access or control over the GPU 305. Direct access or control,in this context, can be allocated to a virtual machine 232B by ahypervisor 202 or control program 220 when the hypervisor 202 or controlprogram 220 allocates direct access to the GPU 305 to the virtualmachine 232B executing on the computer 201. The rendering agent 315 canthen forward the three-dimensional draw commands to the GPU 305 forrendering, and upon receiving the rendered three-dimensional image, canforward the three-dimensional image back to the agent executing on thevirtual machine 232C. In some embodiments, the virtual machine 232C canthen transmit the rendered three-dimensional image to another physicalcomputer 340 remotely located from the computer 201 that includes theGPU 305 and executes the virtual machines 232B, 232C.

The graphics driver 320, in some embodiments, can be a modified graphicsdriver 320 able to intercept two or three-dimensional draw commands uponbeing denied access to a GPU 305. For example, in some embodiments, thegraphics driver 320 can intercept a two or three-dimensional drawcommand and request access to a GPU 305. In response, the graphicsdriver 320 can receive a notification denying access to the GPU 305 tothe graphics driver 320. Upon receiving such a notification, thegraphics driver 320 can then forward the intercepted draw commands toanother virtual machine having access to the GPU 305. In someembodiments, the graphics driver 320 can further be modified to includea hooking mechanism such as any hooking mechanism able to intercept twoor three-dimensional draw commands and forward them to a GPU 305 forrendering or forward them to another virtual machine 232. The hookingmechanism can be a DIRECT3D hooking mechanism. In other embodiments, thegraphics driver 320 can be modified to include a filter that canintercept two or three-dimensional draw commands and forward them to aGPU 305 for rendering or forward them to another virtual machine 232. Instill other embodiments, the graphics driver 320 can be modified toinclude a file system driver, a mini-driver, a mini-filter, a filesystem filter driver or a user-mode hooking mechanism. The graphicsdriver 320, in some embodiments, can be a kernel mode hooking mechanism,e.g. a kernel mode driver, for intercepting two or three-dimensionaldraw commands and forwarding the intercepted commands to a GPU 305 forrendering or to another virtual machine 232. In one embodiment, thegraphics driver 320 can be executed by a guest operating system 230 orwithin the context of a guest operating system 230. The graphics driver320 can communicate with other virtual machines 232 or componentsexecuting on other virtual machines 232 via an intra-virtual-machinenetwork or other intra-virtualization-environment messaging system.While FIG. 3 illustrates a single graphics driver 320 included in eachvirtual machine 232, in other embodiments the virtual machines 232 caninclude any number of graphics drivers.

In one embodiment, the graphics driver 320 can intercept two orthree-dimensional draw commands generated or issued by three-dimensionalapplications 330. In other embodiments, the graphics driver 320 canintercept two or three-dimensional draw commands generated or issued byany application executing on the virtual machine 232 included the guestoperating system 230. The graphics driver 320, in still otherembodiments, can retrieve two or three-dimensional draw commands from astorage repository or buffer. The graphics driver 320 can execute in auser or kernel mode.

The virtual machines 232 can execute any number of three-dimensionalapplications 330. The three-dimensional application 330 can be anyapplication that generates three-dimensional draw commands. Thisapplication output can include any combination of bitmaps, images, anddrawing directives. In some embodiments, the three-dimensionalapplication 330 can generate three-dimensional draw commands able to beinterpreted by any of the following graphics libraries: RECT3D; OGRE;OPENGL; QUESA; TEDDY3D; MESA3D; or any other graphics library that canbe used to render two or three-dimensional images. While FIG. 3illustrates three-dimensional applications 330, the applications 330 cangenerate both three-dimensional and two-dimensional draw commands. Inother embodiments, the virtual machines 232 can further executetwo-dimensional applications.

In some embodiments, the three-dimensional applications 330 cancommunicate with the graphics driver, rendering agent 315 and remotingagent 325. The three-dimensional applications 330 can communicate withthe graphics driver 320 to forward or transmit draw commands to thegraphics driver 320. In other embodiments, the graphics driver 320 canintercept draw commands generated by the three-dimensional application330. The rendering agent 315, in some embodiments, can functionsimilarly to the graphics driver 320 such that the rendering agent 315can intercept draw commands generated by the three-dimensionalapplications 330. In other embodiments, the rendering agent 315 canforward rendered images to the three-dimensional application 330 fordisplay within the context of the application's output window. Theremoting agent 325 can communicate with the three-dimensionalapplication 330 to forward user commands to the application 330, andintercept draw commands for transmission to a remote computer 340.

In some embodiments, the three-dimensional application 330 is acomponent of an operating system. In other embodiments, thethree-dimensional application 330 is the guest operating system 230. Thethree-dimensional application 330, in one embodiment, is a windowsmanager in the WINDOWS VISTA operating system and uses Direct3D commandsto render desktop graphics. in some embodiments, the three-dimensionalapplication 330 generates commands according to an applicationsprogramming interface (such as, by way of example, and withoutlimitation, a Direct3D API.) In another embodiment, the application 330transmits the command to a user-mode runtime component (such as adynamically linked library component within an operating system.) Instill another embodiment, the user-mode runtime component transmits thecommand to a user-mode graphics processing unit driver (e.g., a softwareapplication for transmitting commands to a physical graphics card). Instill even another embodiment, the user-mode graphics processing unitdriver transmits the command to a kernel-mode runtime component (such asanother dynamically linked library component within the operatingsystem). In yet another embodiment, the kernel-mode runtime componenttransmits the command directly or indirectly to the GPU 305. Forexample, the kernel-mode runtime component may transmit the card to aminiport driver for forwarding to the GPU 305 or to the hypervisor 202for modification prior to forwarding to the GPU 305.

The virtual machines 232 can further execute a rendering agent 315. Insome embodiments, the guest operating system 230 executes the renderingagent 315, in other embodiments the rendering agent 315 executes withinthe context of the guest operating system 230. The rendering agent 315,in some embodiments, can receive draw commands from a remote virtualmachine and forward those draw commands to the GPU 305 for rendering.For example, the rendering agent 315 can execute on virtual machine A232B and receive draw commands from virtual machine B 232C. These drawcommands can be generated by an application, e.g. a three-dimensionalapplication 330, executing on virtual machine B 232C. In someembodiments, the graphics driver 320 or another application interceptsthe draw commands generated by an application executing on virtualmachine B 232C and forwards the draw commands to the rendering agent 315executing on virtual machine A 232B. Upon receiving the draw commandsfrom the remote virtual machine, e.g. virtual machine B 232C, therendering agent 315 can transmit or forward the draw commands to the GPU305 for rendering. In some embodiments, the rendering agent 315 canforward the draw commands together with an instruction instructing theGPU 305 to render an image or image file from the accompanying drawcommands. In still other embodiments, the rendering agent 315 can storethe draw commands in a buffer or storage repository and transmit thedraw commands to the GPU 305 when the GPU 305 is available.

In one embodiment, the control virtual machine 232 or another virtualmachine designated as a dedicated rendering virtual machine can be givenexclusive and direct access to the GPU 305. In that embodiment, aninput/output memory management unit (IOMMU) can be used to manage accessto the GPU 305. An IOMMU can be an application, hardware component orcombination application/hardware component that communicates with thehypervisor 202 to further virtualize access to the physical hardwareincluded in the hardware layer 210. In some embodiments, the hypervisor202 can communicate with the IOMMU before transmitting draw commands andinstructions to the GPU 305. Exemplary communication includesinstructions to modify a destination physical memory addressesresponsive to communications with the IOMMU.

In some embodiments, the system illustrated in FIG. 3 can be used torender three-dimensional images from three-dimensional drawing or drawcommands. In other embodiments, the system illustrated in FIG. 3 can beused to render two-dimensional images from two-dimensional drawing ordraw commands.

Illustrated in FIG. 4A is a method 400 for rendering a three-dimensionalimage using virtual machines. In one embodiment, a virtual machine 232or remoting agent 325 executing on a virtual machine 232 receives arequest for three-dimensional application output (Step 402). Theresulting three-dimensional draw commands generated by thethree-dimensional application are intercepted by a graphics driver 320(Step 404). Upon determining that the virtual machine 232 does not havepermission to access the GPU 305, the graphics driver 320 identifies arendering agent 315 or virtual machine 232 having control of the GPU 305(Step 406), and forwards the intercepted draw commands to a renderingagent 315 executing on the identified virtual machine 232 (Step 408).The rendering agent 315 forwards the draw commands to the GPU 305,receives the rendered image from the GPU 305 (Step 410) and forwards therendered image to the requesting virtual machine (Step 412).

Further referring to FIG. 4A, and in more detail, in one embodiment avirtual machine 232 receives a request or three-dimensional applicationoutput (Step 402). In some instances this request can be generated by auser of a remote computer 340, or an application executing on the remotecomputer 340. A remoting application 335 executing on a remote computer340 can forward the request to the virtual machine 232, or in someembodiments can issue the request in response to receiving a request forapplication output. The request, in some embodiments, can include: anidentifier that identifies the application that generates the requestedapplication output; a virtual channel and/or a destination address forthe remote computer 340; a user identifier; or any other informationthat can be used to determine what application output should betransmitted to the requesting computer. While in some embodiments thevirtual machine 232 receives the request, in other embodiments aremoting agent 325 executing on the virtual machine 232 receives therequest. Still other embodiments include any application or object onthe virtual machine 232 able to receive the request.

In response to receiving the request for application output, the virtualmachine 232 or remoting agent 325 can request the graphics output of thethree-dimensional application identified in the request. In someembodiments, requesting the graphics output can include requesting thethree-dimensional images from a desktop manager or other applicationthat manages the rendering of desktop images. In response to a requestfor graphics output, a graphics driver 320 may interceptthree-dimensional draw commands generated by the application (Step 404).Intercepting the three-dimensional draw commands can include hookinginto a call made by the application to a GPU 305, or hooking into a callmade by the application to the guest operating system 230 to render thedraw commands generated by the application. In some embodiments, therequest for application output can be a request for application outputalready generated by an application. In these embodiments, the virtualmachine 232 or remoting agent 325 can retrieve the three dimensionaldraw commands from a graphics buffer or other storage repository storingdraw commands and images generated by the application identified in therequest.

Upon intercepting the draw or drawing commands generated by theapplication 330, the graphics driver 320 can identify a rendering agent325 or virtual machine 232 that has control over the GPU 305 (Step 406).In some embodiments, the graphics driver 320 can identify a renderingagent 325 or virtual machine 232 having control of the GPU 305 byreceiving a notification in response to requesting access to the GPU305. In other embodiments, the graphics driver 320 can identify arendering agent 325 or virtual machine 232 having control or directaccess to the GPU 305 in response to transmitting draw commands and/or arendering instruction to the GPU 305. The response or notification canbe issued by the hypervisor 202, the control program 220 or anotherapplication able to identify the virtual machine 232 or entity that hasdirect access to the GPU 305 and therefore controls the GPU 305. In someembodiments, the method 400 may not include the step of identifying arendering agent 315 having direct access to the GPU 305. Instead, thehypervisor 202 or control program 220 can redirect any renderingrequests issued by any virtual machine other than the virtual machinehaving control over the GPU 305 to the virtual machine having controlover the GPU 305. In these embodiments, the graphics driver 3320 and/orvirtual machine 232 that issued the rendering request may not receive anotification identifying the entity that controls the GPU 305.

In embodiments where the graphics driver 320 obtains information aboutthe virtual machine 232 or rendering agent 315 that controls the GPU305, the graphics driver 320 can forward the drawing commands to thecontrolling entity (Step 408). This controlling entity can be acontrolling virtual machine 232, a controlling guest operating system230 or a controlling rendering agent 315. A controlling entity is anyentity that has direct access to the GPU 305. In some embodiments, thegraphics driver 320 can forward the three-dimensional drawing commandsto the controlling entity via a virtual-machine-to-virtual-machinecommunication protocol. In other embodiments, the graphics driver 320can forward the drawing commands to the controlling entity using anetwork. In still other embodiments, the graphics driver 320 may notforward the drawing commands to the controlling entity. In theseembodiments, the hypervisor 202 or control program 220 may intercept therendering request and redirect it to the entity that has control or thathas a lock on the GPU 305.

The rendering agent 315 having control over the GPU 305 can receive therendered image from the GPU 305 (Step 410) and forward the renderedimage to the virtual machine that requested the image (Step 412). Forexample, the virtual machine 232B that has direct access to the GPU 305,e.g. the controlling entity, can receive the rendered image from the GPU305 once the GPU 305 has rendered the image from the drawing commandsgenerated by an application executing on the virtual machine 232C thatdoes not have direct access to the GPU 305, e.g. a non-controllingentity. In some instances a rendering agent 315 executing on the controlvirtual machine 232B retrieves the rendered image from a buffer to whichthe GPU 305 wrote or saved the rendered image. Upon receiving orretrieving the rendered image, the control virtual machine 232B arendering agent 315 executing on the control virtual machine 232B oranother application executing on the control virtual machine 232B cantransmit the rendered image to the non-controlling virtual machine 232Cexecuting the application that generated the drawing commands from whichthe image was rendered.

In some embodiments, the method 400 can further include transmitting therendered image from the non-controlling virtual machine 232C to a remotecomputer 340 that requested the application output represented by therendered image. In these embodiments, the rendered image represents animage generated from draw commands generated by an application executingon the non-controlling virtual machine. Thus, the rendered imagerepresents the application output requested by the remote computer 340.The non-controlling virtual machine 232 or a remoting agent 325executing on the non-controlling virtual machine can transmit therendered image to the remote computer 340 in response to the remotecomputer's request for application output. In some embodiments, therendered image can be transmitted over a virtual channel establishedbetween the computer 201 and remote computer 340. Thus, the applicationoutput is generated by an application 330B executing on thenon-controlling virtual machine 232C, transmitted to a rendering agent315A executing on the controlling virtual machine 232B, rendered by theGPU 305 to generate an image, the rendered image is transmitted back tothe non-controlling virtual machine 232C and the rendered image isforwarded to the remote computer 340.

Illustrated in FIG. 4B is one embodiment of a method 450 for requestingaccess to the GPU 305. In this method 450, a graphics driver 320 orvirtual machine 232 can request to access the GPU 305 (Step 452). Whenit is determined that another entity has a lock on the GPU 305, therequesting entity can receive a notification denying access to the GPU305 (Step 454). The requesting entity can then forward the drawingcommands to the rendering agent 315 (Step 456).

Further referring to FIG. 4B, and in more detail, in one embodiment agraphics driver 320, a virtual machine 232, a guest operating system 230or an application 330 can request access to the GPU 305 (Step 452).Requesting access to the GPU 305 can include sending drawing commands(e.g. two-dimensional or three-dimensional draw commands) to the GPU 305for rendering. This request can further include an instruction to renderthe transmitted draw commands. In other embodiments, the request caninclude a request to access the GPU 305 and transmit draw commands tothe GPU 305 for rendering. In still other embodiments, the request canbe a request to obtain a lock on the GPU 305 so that the requestingentity can control and exclusively access the GPU 305. A request toaccess the GPU 305, in some embodiments, can be received by a hypervisor202, a control program 220 or any other application or object able tomanage access to the GPU 305. In some embodiments, the request caninclude a virtual machine identifier, a user identifier, or informationabout the type of drawing commands to be forwarded to the GPU 305 forrendering (e.g. metadata indicating three-dimensional rendering will berequested.)

The requesting entity (e.g. the graphics driver 320, virtual machine232, guest operating system 230 or application 330) can receive aresponse or notification denying access to the GPU 305 (Step 454). Insome embodiments, the requesting entity may not receive a notificationdenying access, but rather the hypervisor 202 or control program 220 maydeny access and forward the draw commands to an entity that has a lockon the GPU 305. In these embodiments, the requesting entity may notreceive a notification that the requesting entity was denied access.Instead any draw commands forwarded to the GPU 305 by the requestingentity are redirected to a virtual machine 232 that has direct access,e.g. control, over the GPU 305. In one embodiment, the denialnotification can include the address, identifier or name of the entitythat has a lock on the GPU 305. In other embodiments, the denialnotification can take the form of an error message indicating that theGPU 305 is not available. In some embodiments, the virtualizationenvironment may be configured to provide a control virtual machine 232Awith exclusive access to the GPU 305. In these embodiments, anyrendering request issued by a virtual machine 232B-232D that is not thecontrol virtual machine 232A, may be redirected to a rendering agent 315or other application executing on the control virtual machine 232A.Other applications can include the control program 220 or a graphicsdriver 320 executing on the control virtual machine 232A. In theseembodiments, the control virtual machine 232A can be characterized asany virtual machine 232 executing a control program 220 that managesother virtual machines 232 within the virtualization environment 302.

Upon receiving the denial notification, the requesting entity canforward the draw commands to the entity that has a lock on the GPU 305(Step 456). In some embodiments, this entity can be a rendering agent315 that has control over the GPU 305. Control over the GPU 305 caninclude having direct and exclusive access to the GPU 305. In otherembodiments, the requesting entity can forward the draw commands to therendering agent 315 of the virtual machine 232 that has control over theGPU 305.

Illustrated in FIG. 5A is one embodiment of a method 500 for allocatingaccess to a GPU 305 to a virtual machine 232. A hypervisor 202 orcontrol program 220 allocates direct access to a GPU 305 to a virtualmachine 232 (Step 502). A rendering agent 315 receives draw commands(e.g. two-dimensional or three-dimensional) from another virtual machine232 (Step 504) and forwards those draw commands to the GPU 305 (Step506). The rendering agent 315 then receives a rendered image from theGPU 305 (Step 508) and forwards the rendered image to the virtualmachine 232 (Step 510).

Further referring to FIG. 5A, and in more detail, in some embodiments ahypervisor 202 can allocate direct access to a GPU 305 to a virtualmachine 232 (Step 502). While in some embodiments, the hypervisor 202can allocate direct access in other embodiments a control program 220can allocate direct access. The access to the GPU 305 can be allocatedor given to a virtual machine 232, a rendering agent 315 executing on aparticular virtual machine 232, a graphics driver 320 executing on aparticular virtual machine 232 or any other application or objectexecuting on a particular virtual machine 232. In some embodiments,direct access can include exclusive access to the GPU 305 such that theGPU 305 renders only draw commands generated by applications 330 oroperating systems 230 executing on the virtual machine 232 allocateddirect access to the GPU 305. Exclusive access can include preventingother virtual machines 232 from accessing the GPU 305 and having the GPU305 render images from draw commands generated by those other virtualmachines 232.

The rendering agent 315 executing on the virtual machine 232 that wasgiven direct access to the GPU 305 (e.g. the control rendering agent315) can receive draw commands from other virtual machines 232 (Step504). These draw commands can be generated by applications 330 oroperating systems 230 executing on other virtual machines 232 (e.g.virtual machines not given control or direct access to the GPU 305). Insome embodiments, a graphics driver 320 or rendering agent 315 executingon the other virtual machines 232 can forward the draw commands to thecontrol rendering agent 315. The commands may be forwarded to thecontrol rendering agent 315 for rendering.

Upon receiving the draw commands, the control rendering agent 315 canforward the received draw commands to the GPU 305 (Step 506). In someembodiments, the rendering agent 315 can just forward the draw commands,in other embodiments the control rendering agent 315 can forward thedraw commands together with an instruction to render the GPU 305. Uponreceiving the draw commands and/or the rendering instruction, the GPU305 can render an image from the draw commands. In some embodiments theGPU 305 can send the control rendering agent 315 the rendered image. Inother embodiments, the GPU 305 can store the rendered image in a bufferor other storage repository.

The rendering agent 315 can then receive the rendered image from the GPU305 (Step 508). In some embodiments, the control rendering agent 315 canretrieve the rendered image from a buffer or other storage repository.Upon receiving the rendered image, the control rendering agent 315 canforward the rendered image to the virtual machine 232 that forwarded thedraw commands (Step 510).

Illustrated in FIG. 5B is one embodiment of a method 550 for allocatingdirect access to the GPU 305 to a requesting entity. A hypervisor 202 orcontrol program 220 receives a request from a virtual machine 232,rendering agent 315 or graphics driver 320 for access to the GPU 305(Step 552). A decision is then made as to whether the requesting virtualmachine 232 (or requesting entity executing on a virtual machine) has alock on the GPU 305 (Step 554). This decision can be made by thehypervisor 202 or the control program 220. When it is decided that therequesting virtual machine 232 has a lock, the hypervisor 202 or controlprogram 220 permits the GPU 305 to render the image (Step 556). When itis determined that the requesting virtual machine 232 does not have alock on the GPU 305, the hypervisor 202 or control program 220 furtherdetermines whether there exists a virtual machine 232 that has a lock onthe GPU 305 (Step 558). When it is determined that there does existanother virtual machine 232 (i.e. other than the requesting virtualmachine) that has a lock on the GPU 305, then the hypervisor 202 orcontrol program 220 sends a notification to the requesting virtualmachine 232 denying access to the GPU 305 (Step 562). When it isdetermined that there does not exist another virtual machine 232 thathas a lock on the GPU 305, the hypervisor 202 or control program 220 canallocate direct and exclusive access to the GPU 305 to the requestingvirtual machine 232 (Step 560), and the GPU 305 can render an image fromthe draw commands forwarded by the requesting virtual machine 232 (Step556).

One example of using virtual machines to render three-dimensional drawcommands generated by an application can include allocating access tothe GPU 305 to a control virtual machine 232B. The control virtualmachine can be a control virtual machine 232A executing a controlprogram 220 or another virtual machine 232B. The hypervisor 202 canallocate access to the GPU 305, where the GPU 305 resides on the samephysical computer 201 that executes the control virtual machine 232B. Insome embodiments, the control virtual machine 232B can execute anon-paravirtualized operating system 230. An agent or graphics driver320 executing on another virtual machine 232C, e.g. a virtual machine232C that doesn't have direct access to the GPU 305, interceptsthree-dimensional draw commands generated by an application 330Bexecuting on the non-control virtual machine 232C. In some embodiments,the agent 320B can directly forward the three-dimensional draw commandsto a rendering agent 315A executing on the control virtual machine 232B.In other embodiments the agent 320B can request access to the GPU 305and upon receiving a notification denying access to the GPU 305, theagent 320B can redirect the three-dimensional draw commands to therendering agent 315A. Upon receiving the three-dimensional drawcommands, the rendering agent 315A can forward the commands to the GPU305 which can generate an image from the three-dimensional draw commandsupon receiving the commands. In some embodiments, the GPU 305 cantransmit the rendered image to the rendering agent 315A, in otherembodiments the rendering agent 315A can retrieve the image from abuffer. Upon retrieving or receiving the rendered image, the renderingagent 315A can forward the rendered image to the agent 320B executing onthe non-control virtual machine 232C. The non-control virtual machine232C, in response to receiving the rendered image, can transmit therendered image to a remote, physical computer 340 over a communicationchannel established over a network connecting the computer 201 andremote computer 340. In this example, the agent 320B can be a graphicsdriver or another application executing on the non-control virtualmachine 232C. Another application could be a redirection agent orhooking application able to intercept drawing commands generated byapplications executing on the virtual machine 232C.

While certain exemplary embodiments have been described and shown in theaccompanying drawings, it is to be understood that such embodiments aremerely illustrative of and not restrictive on the methods and systemsdescribed herein. Additionally, it is possible to implement the methodsand systems described herein or some of its features in hardware,programmable devices, firmware, software or a combination thereof. Themethods and systems described herein or parts of the methods and systemsdescribed herein may also be embodied in a processor-readable storagemedium or machine-readable medium such as a magnetic (e.g., hard drive,floppy drive), optical (e.g., compact disk, digital versatile disk,etc), or semiconductor storage medium (volatile and non-volatile).

1-20. (canceled)
 21. A method for rendering draw commands generated by an application, the method comprising: intercepting, by an agent executing on a first virtual machine, draw commands generated by an application on the first virtual machine; receiving, by the agent, a notification denying the agent access to a graphics processing unit, the notification identifying a second virtual machine allocated with the graphics processing unit; redirecting, by the agent, the intercepted draw commands to a rendering agent on the second virtual machine; rendering, by the graphics processing unit, an image based on the redirected draw commands; and forwarding, by the rendering agent on the second virtual machine, the rendered image from the graphics processing unit to the agent executing on the first virtual machine.
 22. The method of claim 21, wherein intercepting the draw commands comprises intercepting three-dimensional draw commands generated by the application on the first virtual machine, the first virtual machine executing a non-paravirtualized operating system.
 23. The method of claim 21, further comprising transmitting, from the first virtual machine responsive to receiving the rendered image from the rendering agent, the rendered image to a second physical computer over a communication channel established between the second physical computer and a first physical computer hosting the first virtual machine.
 24. The method of claim 23, wherein transmitting the rendered image to the second physical computer further comprises transmitting, by a remoting agent executing on the first virtual machine, the rendered image to the second physical computer.
 25. The method of claim 21, further comprising transmitting, by the rendering agent responsive to receiving the redirected draw commands, the redirected draw commands to the graphics processing unit.
 26. The method of claim 21, further comprising receiving, by the first virtual machine, a request from a second physical computer for application output generated by the application executing on the first virtual machine.
 27. The method of claim 26, further comprising transmitting the rendered image to the second physical computer in response to receiving the request from the second physical computer.
 28. The method of claim 21, further comprising allocating to the second virtual machine, by a hypervisor executing on a first physical computer executing the first virtual machine and the second virtual machine, direct access to the graphics processing unit of the first physical computer.
 29. The method of claim 21, further comprising requesting, by the agent, access to the graphics processing unit responsive to intercepting the draw commands.
 30. The method of claim 21, wherein redirecting the intercepted draw commands further comprises redirecting the intercepted draw commands responsive to receiving the notification, the notification comprising a message from a hypervisor denying the first virtual machine access to the graphics processing unit.
 31. A system for remotely rendering draw commands generated by an application, the system comprising: a graphics processing unit; and an agent executing on a first virtual machine of a first physical computer, the agent configured to: intercept draw commands generated by an application executing on the first virtual machine; receive a notification denying the agent access to the graphics processing unit, the notification identifying a second virtual machine allocated with the graphics processing unit; and redirect the intercepted draw commands to a rendering agent on the second virtual machine; wherein the graphics processing unit is configured to render an image based on the redirected draw commands, and the rendering agent is configured to forward the rendered image from the graphics processing unit to the agent executing on the first virtual machine.
 32. The system of claim 31, wherein the agent is configured to intercept three-dimensional draw commands generated by the application on the first virtual machine, the first virtual machine executing a non-paravirtualized operating system.
 33. The system of claim 31, further comprising a remoting agent executing on the first virtual machine, the remoting agent configured to transmit, responsive to receiving the rendered image from the rendering agent, the rendered image to a second physical computer over a communication channel established between the second physical computer and the first physical computer.
 34. The system of claim 33, wherein the remoting agent is configured to transmit the rendered image to the second physical computer responsive to receiving the rendered image from the rendering agent.
 35. The system of claim 31, wherein the rendering agent is configured to transmit the draw commands to the graphics processing unit responsive to receiving the draw commands from the agent.
 36. The system of claim 31, wherein the first virtual machine receives a request from a second physical computer for application output generated by the application executing on the first virtual machine.
 37. The system of claim 36, further comprising a remoting agent executing on the first virtual machine, the remoting agent configured to transmit the rendered image to the second physical computer in response to receiving the request from the second physical computer.
 38. The system of claim 31, further comprising a hypervisor executing on the first physical computer, the hypervisor configured to allocate to the second virtual machine direct access to the graphics processing unit of the first physical computer.
 39. The system of claim 31, wherein the agent is further configured to request access to the graphics processing unit responsive to intercepting the draw commands.
 40. The system of claim 31, wherein the agent is configured to redirect the intercepted draw commands responsive to receiving the notification, the notification comprising a message from a hypervisor denying the first virtual machine access to the graphics processing unit. 